Scalable tenant networks

ABSTRACT

Template-driven locally calculated policy updates for virtualized machines in a datacenter environment are described. A central control and monitoring node calculates and pushes down policy templates to local control and monitoring nodes. The templates provide boundaries and/or a pool of networking resources, from which the local control and monitoring node is enabled to calculate policy updates for locally instantiated virtual machines and containers.

PRIORITY APPLICATIONS

This patent application is a divisional application of, and claimspriority to, U.S. Non-Provisional patent application Ser. No.16/706,456, entitled “SCALABLE TENANT NETWORKS”, filed on Dec. 6, 2019,which is incorporated herein by reference in its entirety. This patentapplication is a continuation application of, and claims priority to,co-pending, commonly owned U.S. Non-Provisional patent application Ser.No. 16/054,638, entitled “SCALABLE TENANT NETWORKS”, filed on Aug. 3,2018, which is incorporated herein by reference in its entirety. U.S.Non-Provisional patent application Ser. No. 16/054,638 claims priorityto U.S. Non-Provisional patent application Ser. No. 15/859,247 (U.S.Pat. No. 10,044,525), entitled “SCALABLE TENANT NETWORKS,” filed on Dec.29, 2017, which claims priority to U.S. Non-Provisional patentapplication Ser. No. 15/075,049, filed Mar. 18, 2016, and entitled“SCALABLE TENANT NETWORKS”, which is also incorporated herein byreference in its entirety. U.S. Non-Provisional patent application Ser.No. 15/075,049 claims priority to U.S. Provisional Patent ApplicationNo. 62/267,664, filed Dec. 15, 2015, and entitled “SCALABLE TENANTNETWORKS”, which is also incorporated herein by reference in itsentirety.

BACKGROUND

Managing change in a datacenter has always been difficult. In the earlydays, people had to manually connect cables and update configurationfiles. Automation technologies and techniques such as Dynamic HostConfiguration Protocol (DHCP), dynamic Domain Name System (DNS)registrations, Virtual Local Area Networks (VLANs), and switchmanagement scripts lightened the load. Datacenters have experiencedmassive growth. To enable more flexibility, a new generation oftechnologies and techniques have been created such as networkvirtualization and network functions virtualization (NF V). These createflexibility to manage this new hyperscale datacenter. Recently, anothervariable has been added to the equation: containers and nested virtualmachines (VMs). These developments abstract the tenant's network fromthe hoster's physical network and virtual network. This flexibility andmulti-tenant isolation will add complex management overhead andultimately, scalability.

BRIEF SUMMARY

This Summary is provided in order to introduce simplified concepts ofthe present disclosure, which are further described below in theDetailed Description. This summary is not intended to identify essentialfeatures of the claimed subject matter, nor is it intended for use indetermining the scope of the claimed subject matter.

Examples of the present disclosure provide template-driven local policyupdates for virtualized machines in a datacenter environment. A centralcontrol and/or monitoring node (e.g., a “central controller”) calculatesand pushes down policy templates to local control and/or monitoringnodes (e.g., the “local controllers”). The templates provide boundariesand/or a pool of networking resources, from which the local controland/or monitoring node is enabled to calculate policy updates forlocally instantiated virtual machines and containers. The local controland monitoring node is configured to monitor the allotted resourceutilization, and request from the central controller additional orupdated templates and resources for those virtual machines andcontainers within its control.

BRIEF DESCRIPTION OF THE DRAWINGS

The Detailed Description is set forth with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical items.

FIG. 1 is a block diagram that illustrates an example environment fortemplate-driven local policy updates for virtualized machines in adatacenter environment.

FIG. 2 is a block diagram that illustrates an example environment fordiscoverability in a dynamic datacenter environment.

FIG. 3 is a block diagram of an example computing system usable toimplement various aspects of the present disclosure.

FIG. 4 illustrates a flow chart depicting an example process fortemplate-driven local policy updates for virtualized machines in adatacenter environment.

FIG. 5 illustrates a flow chart depicting an example process formonitoring resource utilization in the datacenter network and requestingmore resources.

DETAILED DESCRIPTION

As noted above, the added flexibility and multi-tenant isolation willadd complex management overhead and ultimately, scalability. Embodimentsof the present application provide a way to manage this flexibility, inpart by providing template-driven local policy updates for virtualizedmachines in a datacenter environment. A central control and monitoringnode (referred to herein as “central controller”) calculates and pushesdown policy templates to local control and monitoring nodes (referred toherein as “local controllers”). The templates provide boundaries and/ora pool of networking resources, from which the local controller isenabled to calculate policy updates for locally instantiated virtualmachines and containers. The local controller is configured to monitorthe allotted resource utilization, and request from the centralcontroller additional or updated templates and resources for thosevirtual machines and containers within its control. A typical datacenterdeployment will have many central controllers and many localcontrollers. Some embodiments may support a 1:1 relationship betweencentral controller instances and local controller instances. Otherembodiments may have fewer central controllers servicing many more localcontrollers.

Containers are an isolated, resource-controlled, and portable operatingenvironment. A software application can run in a container withoutaffecting the rest of the system and without the system affecting theapplication. Containers may be specific to one application, or mayprovide the runtime for multiple applications, providing additionaloperating system functions and services (e.g. input/output functions,security, etc.).

Containers and nested virtual machines (VMs) (e.g., virtual machinesexecuting within another virtual machine) introduce additional levels ofabstraction to the datacenter, thereby increasing the complexity of thedatacenter environment, including the networking infrastructure. This isespecially true when new VMs or containers are instantiated, or whenexisting containers or VMs are moved.

To use just one example, a physical host has a physical IP address, avirtual machine executing within the physical host has a virtual IPaddress, and the nested VMs or containers executing within the virtualmachine have their own IP addresses. These IP addresses may be ondifferent subnets. Network Virtualization Functions (NVF) in thephysical host may be used to encapsulate and de-encapsulate(“encap/decap”) packets or to translate addresses from one layer ofabstraction to another, and between physical and virtual layers. Thus,policies to perform such encap/decap or network address translation areupdated at multiple levels each time a nested VM or container isinstantiated or moved, thereby increasing the complexity of policyupdates. Also, a container may be instantiated for as little as 10-90milliseconds, just long enough to execute a script, before it is takendown. But utilizing a centralized network control and monitoring node tocalculate and push networking policy updates for a newly instantiatedvirtual machine or container may take on the order of seconds or minutesto update the networking infrastructure. This may be too long for acontainer that may need to be reachable within a few milliseconds ofbeing instantiated, and which may exist for only a few millisecondsbefore being deleted.

Embodiments of the present disclosure push policy calculations andupdates to the local host environment, providing both the guidelines andresources that enable local calculation and policy application. Thisincreases the speed with which the policies and networkinginfrastructure configuration can be updated. Templates provideconstraints on the policies that may be calculated, thereby delegatingsome configuration ability to a local controller. This enables thedatacenter networking infrastructure to better scale to large numbers ofdynamically instantiated containers and virtual machines, includingnested VMs and containers, while maintaining predictability, networkstability, and centralized administrative oversight of the network.

Various examples are described below with respect to the figures. Theseexamples are merely exemplary, and embodiments of the present disclosureare not limited to the examples described below and illustrated in thefigures.

FIG. 1 is a block diagram that illustrates an example environment fortemplate-driven locally calculated policy updates for virtualizedmachines in a datacenter environment. Environment 100 includes aplurality of host nodes 102, 104, and 106. The host nodes 102-106include a plurality of virtual machines (VMs), such as the VMs 108 shownin host node 102. The host node 102 is illustrated as having threevirtual machines, but more or fewer virtual machines may be includedwithout departing from the scope of embodiments.

Virtual machine 108-1 includes containers 110 and 112. Although virtualmachine 108-1 is illustrated as having two containers 110 and 112,nested VMs may be instantiated within the virtual machines according toembodiments. Containers 110 and 112 are illustrated in FIG. 1 for thesake of description. Also, the virtual machine 108-1 is illustrated ashaving two containers, but more or fewer containers (or more or fewernested virtual machines) may be included in a virtual machine withoutdeparting from the scope of embodiments. A virtual machine such asvirtual machine 108-1 may include up to 200 or more nested VMs orcontainers in some embodiments.

The host node 102 is illustrated as including a network functionsvirtualization (NFV) module 114. The NFV 114 is configured to providevirtualized networking functions for the host node 102, such as routing,switching, bridging, firewall functions, network address translation(NAT), encap/decap, load balancing, Dynamic Host Configuration Protocol(DHCP), name resolution service (e.g., DNS resolution), and othervirtualized networking functions. Similarly, the virtual machine 108-1includes NFV module 116, which may provide the same, similar, ordifferent virtualized network functions for the virtual machine 108-1.In some embodiments, a NFV module may be run in a virtual machine orcontainer or distributed among multiple virtual machines and containers.

A central controller 118 (e.g., a central control and monitoring node)includes a policy store 120 and a policy calculation module 122. Thepolicy calculation module 122 calculates, based on configuration from anadministrator and/or based on requests from the host nodes and/or localcontroller 124, one or more templates 126. The templates 126 definevarious configurable elements of a networking policy that may becalculated by the local controller 124.

Example configurable policy elements defined by the templates 126include:

1. A pool of resources (e.g., IP addresses, Media Access Control (MAC)addresses, port numbers, and so forth) that may be allocated tocontainers 110 and 112, or other containers or nested virtual machineswithin any of the host nodes 102-106.

2. Customer Addresses (CAs) for one or more routing domain identifiers(RDIDs). An RDID may be a uniform resource identifier (URI), or otherunique identifier that identifies a group of containers 110, 112 ornested VMs that fall within a single domain of control. Where thecontainers 110, 112 or nested VMs are part of a customer tenant networkwithin the datacenter environment 100, one or more RDIDs may be utilizedto identify those containers 110, 112 or nested VMs. The templates 126may specify the CAs (e.g., the public-facing IP addresses) that may beallocated to the containers 110, 112 within an RDID.

3. Load balancer Virtual IP (VIP) to dynamic IP (DIP) address mapping. AVIP is the public IP address of the container or nested VM, which may beallocated from a pool defined in the template. The DIP is the internalIP associated with a VM or container. The templates 126 may designateranges of VIPs and ranges of DIPs that may be mapped for load-balancingor other purposes.

4. Service chain rules. A service chain specifies the networkappliances, such as load-balancers, anti-virus scanners, firewalls,deep-packet inspection servers, and so forth, which packets traverse ontheir way to and from a destination in the datacenter environment 100.The appliances may be virtualized appliances executing as virtualmachines on a host in the environment 100, implemented as stand-alonephysical appliances, or some combination of these. In some embodiments,a MAC forwarding table specifies the MAC addresses and the order of theappliances in the service chain. The templates 126 specify constraintsfor service chain configuration, including permissive service chainelements that the local controller 124 may use, as well as mandatoryservice chain elements that the local controller must specify in apolicy for a nested VM or container.

5. Security policies. Security policies include access control lists(ACLs), firewall rules, and so forth for enforcing security policies.The ACLs may specify a 5-tuple of source port, source address, protocol,destination port, and destination address that define packets that areallowed or denied entry into a network through a network device, such asa firewall. The firewall rules may include such things as whetherpackets are dropped, forwarded, redirected, subjected to stateful orstateless packet inspection, and so forth. The templates 126 may specifyaspects of ACLs, firewall rules, and so forth that may be configured bythe local controller 124, within certain constraints. A template 126 mayspecify a list of firewall rules, ACLs and so forth that may be used ina policy, a range of ports that may be permitted, and so forth. Thetemplate 126 may also enforce the use of certain ACLs and firewallrules, which cannot be altered by the local controller 124. Thus in someembodiments, the templates 126 may include permissive policy elementsthat the local controller 124 may use as well as mandatory policyelements that the local controller must use.

6. Local forwarding tables. Local forwarding tables include thedestinations with which a VM or container is able to communicate. Localforwarding tables may include encap/decap rules, network addresstranslation rules, and so forth that enable the containers 110, 112 tocommunicate with other elements within the datacenter environment 100.The local forwarding tables may specify certain ranges of IP addressesthat may be reachable by a nested VM or container.

The local controller 124 includes a policy calculation module 128, whichdetermines that nested VMs or containers are instantiated or movedwithin a local group of host nodes 102-106 or virtual machines 108 forwhich it has control. The local controller 124 is illustrated as aseparate device from the host nodes 102-106, although in someembodiments it may reside on one or more of the host nodes 102-106, aswell as being a separate, standalone device or server. The policycalculation module 128 utilizes the templates 126 stored within thepolicy store 130 to calculate policies 132. The policies 132 include atleast the configurable policy elements described elsewhere within thisDetailed Description. One or more of the policies 132 may be provided tothe NFV 114, the NFV 116, and the network infrastructure 134 of thedatacenter environment 100 (e.g., including host nodes 102, 104, 106),as shown with the arrows illustrated in FIG. 1. The policies are usableto reconfigure network elements, such as the NFV 114, the NFV 116, andthe network infrastructure 134, to support the various networkingfunctions designated for the containers. For example, the policies maybe usable to configure NAT, encap/decap, firewall rules, ACLs, localforwarding, IP address mapping, load-balancing, and various othernetworking configurations that may be supported in the environment 100.In another example, the policies may be useable and/or tailored toconfigure virtualized network functions instantiated within virtualmachines.

In one example of a configurable policy, host node 102 is illustrated inFIG. 1 having a physical IP address of 10.1.1.1. The VM 108-1 isillustrated having an IP address of 192.168.1.1, and containers 110 and112 are illustrated as having, respectively, IP addresses 1.1.1.1 and1.1.1.2. The policies 132 may specify that a newly instantiatedcontainer (e.g., container 112) is assigned customer IP address 1.1.1.2.In this case, the templates 126-1 may specify that containersinstantiated by the local controller 124 within a certain RDID may beselected from a range of IP addresses that includes 1.1.1.2 (e.g., fromthe 1.1.0.0/16 subnet). The policies 132 may specify not only the IPaddress for the container 112, but also the local forwarding tables,including NAT rules and encap/decap rules, which are enforced by one ormore of the NFV 114 or the NFV 116, security policies and service chainsenforced by the NFV 114 or the NFV 116 as well as resources within thenetwork infrastructure 134, and so forth.

The local capacity monitoring module 136 of the local controller 124monitors the utilization of its pool of resources defined by thetemplates 126. Based on certain predetermined mechanisms, the localcapacity monitoring module 136 may request additional resources from thecentral controller 118. For example, a request for more IP addresses maybe based on a threshold number of IP addresses being allocated (e.g.,80% of IP addresses allocated), based on current trends in IP addressallocation, based on real-time data, and so forth. Similarly, thecentral capacity monitoring module 138 may also monitor utilization ofresources, and assign such resources to the local controller 124 viatemplate updates. By monitoring a local host environment within thelarger datacenter environment, the local capacity monitoring module 136may be able to respond more quickly to depletion of resources than woulda central controller 118.

FIG. 2 is a block diagram that illustrates an example environment 200for discoverability in a dynamic datacenter environment. Environment 200includes host node 202 and host node 204, which may be the same as orsimilar to the host node 102 illustrated in FIG. 1. Host node 202includes a virtual machine 206, in which containers 208 and 210 areinstantiated. An NFV module 212 provides virtualized networkingfunctions to the container 208 and 210, and to the virtual machine 206in conjunction with NFV 214. Similarly, host node 204 includes a virtualmachine 216, in which containers 218 and 220 are instantiated. An NFVmodule 222 provides virtualized networking functions to the container218 and 220, and to the virtual machine 216 in conjunction with NFV 224.

With some containers and nested VMs being instantiated for only shortperiods of time, it could be difficult to identify and reach suchcontainers and nested VMs via the network infrastructure 134 usingconventional methods. Using conventional network discovery mechanismsmay take too long to be practical in a dynamic datacenter environment.IP routes may take a few minutes to update. DNS and other namingservices take up to 30 minutes to update. A tenant administrator (anadministrator of a RDID) may define whether containers in the RDID areto be provided with service availability and may input that into acontainer management policy.

FIG. 2 illustrates two methods for providing faster discoverability.First, for each container 208, 210, 218, 220 that is configured forservice availability, the local control and monitoring node deploys one“IP anchor” container, illustrated in FIG. 2 as the anchor container218. The role of the “IP anchor” container is to act as a proxy and as aNAT server, thereby providing automatic discoverability and networkreachability. All other containers that are deployed on the node or inthe RDID have a default IP route to the “IP anchor” container. The localcontrol and monitoring node then monitors the connectivity and DNSrecords of each of the container services. When the routes and DNSrecords are available, client software will access the containerdirectly. In at least one example, the local control and monitoring nodeis configured to provide discoverability services for a virtual machine,a container, or a virtualized network function that has migrated toanother system.

In various embodiments, the local control and monitoring node can use anIP anchor to ensure connectivity to a virtualized network functioninstantiated within a virtual machine. Moreover, the local control andmonitoring node can monitor and update a policy that is tailored to avirtualized network function and that is associated with the IP anchor.

In some embodiments, mobile IP, such as is described in IETF RFCs 5944,6275, and 4721, and others, may be employed. In these embodiments, thecontainer 208, 210, 218, or 220 may be configured with a permanent homeaddress and a care-of address. A home agent (e.g., the anchor container218), is configured with the care-of address, and acts as a conduit forthe other containers within the network. A tunnel is established betweenthe home agent and the containers. Where a container moves to anotherhost, an anchor container 218 on the other host may act as a foreignagent, which acts a conduit for IP addresses for the container.

In some other embodiments, one or more of the NFVs 212, 214, 222, and224 may be configured to respond to broadcast queries for reachabilityinformation. For example, NFV 214 may be configured to provideconnectivity services for containers or nested VMs having IP addresses1.1.1.0/24 subnet, and the NFV 224 may be configured to provideconnectivity services for containers or nested VMs having IP addressesin the 1.1.2.0/24 subnet. A neighbor list in the host node 202 and inthe host node 204 provides information regarding the host nodeneighbors. In the example illustrated in FIG. 2, the host node 202 andthe host node 204 are both configured with each other in their neighborlists, along with other host nodes.

When a container 208 or 210 on the host node 202 attempts to contact acontainer having an IP address in the 1.1.2.0 subnet, the NFV 214 sendsa broadcast to its neighbors in the neighbor list for the neighborhaving responsibility for the 1.1.2.0 subnet. The NFV 224 responds tothe query with a response indicating that host node 204, at IP address10.1.1.2, is responsible for containers having IP addresses in the1.1.2.0 subnet. The response may indicate the IP address of the virtualmachine 216 (192.168.2.1), so that the encap/decap or NAT may beemployed at the various layers of NFV within the host node 202 to reachthe 1.1.2.0 subnet. Thus, packets from containers 208 or 210 destinedfor one of containers 218 or 220 may be forwarded to the NFV 224, usingeither encap/decap and/or NAT. In some scenarios, if the requestedcontainer has moved to a different location, the NFV 224 may againbroadcast for the next location and forward to the appropriate NFV usingeither encap/decap and/or NAT.

The neighbor list may be provided by the central controller 118, by thelocal controller 124, and so forth. In other embodiments, the centralcontroller 118 and/or the local controller 124 may be configured toprovide a list of hosts and related subnets, so that discovery does notneed to take place to route packets within an RDID or other local hostarea.

In various embodiments, configurable network policy elements of a policytemplate can include sets of network addresses that are looselyassociated with each other. The loose association can be based on apolicy relationship (e.g., a routing relationship) between a firstvirtual machine and a second virtual machine. The policy template can beupdated when growth and/or changes occur within the environment 200. Forexample, if the load increases and more containers are added to virtualmachine 206, more virtual machines (and associated containers) are addedto host node 202, and/or more NFV nodes/modules are configured to handlerouting and network policy enforcement, then a policy template may beassociated with allocating more subnets to the added networkinfrastructure elements. Moreover, as the growth and changes occur, somecontainers may move and/or some containers may be suspended. Themovement and the suspension can be tracked and/or monitored by anassociated IP anchor, and thus, the IP addresses of moving containers orsuspended containers cannot be allocated. Consequently, the policytemplate configured and distributed by the central controller 118 canaccount for IP addresses that cannot be allocated. For example, a policytemplate can provide:

-   -   (i) a first set of IP addresses at a first layer such as the        physical network (e.g., 10.1.1.1 for host node 202 and 10.1.1.2        for host node 204 from subnet 10.1.1.0/24) and an indication        whether an individual IP address is allocated or unallocated,    -   (ii) a second set of IP addresses at a second layer such as the        host NVF network (e.g., subnet 192.168.1.0/24 is associated with        address 10.1.1.1 and 192.168.1.1 is allocated for a specific NFV        212) and an indication whether an individual IP address is        allocated or unallocated, and/or    -   (iii) a third set of IP addresses at a third layer such as the        containers (e.g., subnet 1.1.1.0/24 is associated with address        192.168.1.1) and an indication whether an individual IP address        is allocated or unallocated.

Moreover, an allocated IP address may be a reachable IP address, anunreachable IP address, or a redirected IP address. Since adding a newsubnet may ultimately introduce routing dependencies and/or changedownstream routing, the central controller 118 can track the usage of IPaddresses and potential allocations of IP addresses so that routingchanges can be distributed in time for a particular layer in the aboveaddressing stack to request more IP addresses.

FIG. 3 is a block diagram of an example computing system 300 usable toimplement various aspects of the present disclosure. Computing system300 may be deployed in a shared network environment, including in adatacenter, a cloud computing environment, or other network of computingdevices. In one example configuration, the computing system 300comprises at least one processor 302 and computer-readable media 304.The computing system 300 also contains communication connection(s) 306that allow communications with various other systems. The computingsystem 300 also includes one or more input devices 308, such as akeyboard, mouse, pen, voice input device, touch input device, etc., andone or more output devices 310, such as a display (including atouch-screen display), speakers, printer, etc. coupled communicativelyto the processor(s) 302 and the computer-readable media 304 viaconnections 312.

In the illustrated example, computer-readable media 304 stores operatingsystem(s) 314, which provide basic system functionality to: virtualmachines 316 (which may be the same as or similar to one or more ofvirtual machines 108, 206, and 216), containers 318 (which may be thesame as or similar to one or more of containers 110, 112, 208, 210, 218,and 220), NFV 320 (which may be the same as or similar to one or more ofNFV 114, 116, 212, 214, 222, and 224), policy store(s) 322 (which may bethe same as or similar to one or more of policy store 120 and 130),local policy calculation module 324 (which may be the same as or similarto the local policy calculation module 128), local capacity monitoring326 (which may be the same as or similar to the local capacitymonitoring 136), central policy calculation module 328 (which may be thesame as or similar to the central policy calculation module 122), andcentral capacity monitoring 330 (which may be the same as or similar tothe central capacity monitoring 138).

FIGS. 4 and 5 depict flow diagrams that show example processes inaccordance with various examples. The operations of these processes areillustrated in individual blocks and summarized with reference to thoseblocks. These processes are illustrated as logical flow graphs, eachoperation of which may represent a set of operations that can beimplemented in hardware, software, or a combination thereof. In thecontext of software, the operations represent computer-executableinstructions stored on one or more computer storage media that, whenexecuted by one or more processors, enable the one or more processors toperform the recited operations. Generally, computer-executableinstructions include routines, programs, objects, modules, components,data structures, and the like that perform particular functions orimplement particular abstract data types. The order in which theoperations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order, separated into sub-operations, and/or performed inparallel to implement the processes. Processes according to variousexamples of the present disclosure may include only some or all of theoperations depicted in the logical flow graphs.

FIG. 4 illustrates a flow chart depicting an example process 400 fortemplate-driven local policy updates for virtualized machines in adatacenter environment. At 402, a central administrator allocates thenetwork policy for a given tenant (e.g., an RDID or other local hostdomain) via a central controller, such as the central controller 118.

At 404, the local controller communicates with the central controller todeclare that a host node (such as the host node 102) or a VM (such asthe VMs 108) hosts a certain number of containers or nested VMs. Thelocal controller also declares the policies that need to be provided forthose containers or nested VMs. Example policy elements that the localcontroller declares includes one or more IP subnets from which IP endpoints of the containers are to be allocated.

At 406, the central controller distributes policy templates to the localcontroller, and thus, the local controller receives the policy templatefrom the central controller. Various policy elements and configurationconstraints may be specified within the policy templates, including oneor more pools of resources (e.g., IP addresses, Media Access Control(MAC) addresses, port numbers, and so forth) that may be allocated tocontainers, customer Addresses (CAs) for one or more routing domainidentifiers (RDIDs), load balancer Virtual IP (VIP) to dynamic IP (DIP)address mapping, service chain rules, security policies, localforwarding tables, and so forth.

At 408, the local controller calculates the network virtualizationpolicy without the need to contact the central controller, includingallocating IP addresses based on the policy templates. Calculating thepolicies includes calculating the parameters of the policy based on theresources and constraints included in the policy templates.

At 410, the local controller distributes the policies to networkinfrastructure elements and to the host nodes. The policies are used toconfigure the network infrastructure elements and the host nodes toprovide the appropriate networking connectivity for the nested VMs orcontainers.

FIG. 5 illustrates a flow chart depicting an example process 500 formonitoring resource utilization in the datacenter network and requestingmore resources. At 502, the local controller monitors the resourceutilization within the RDID or other local hosting domain. In oneexample, a percent of resources allocated is determined (e.g., a percentof IP addresses allocated). In another example, trends or predictionsregarding future resource utilization is determined. Such trends may bebased on real-time data, historical data, current utilization, trafficstatistics, or other data that may indicate future resourcerequirements.

At 504, the local controller determines, based on the resourceallocation monitoring, to request additional resources. The localcontroller may transmit a request to a central controller to requestadditional resources.

At 506, the local controller receives, in response to the request formore resources, one or more updated policy templates. The policytemplates may indicate new constraints on policy elements that the localcontroller may configure, including an increase in a range of resourcesthat the local controller may allocate to nested VMs or containers,including for example an expanded or otherwise larger range of IPaddresses to allocate.

According to various non-limiting examples, the computing systemsdescribed herein includes one or more devices, such as servers, storagedevices, tablet computers, laptops, desktop computers, gaming consoles,media players, mobile phones, handheld computers, wearable devices,smart appliances, networking equipment, kiosk devices, and so forth. Inone example configuration, the computing systems comprise at least oneprocessor. The computing systems also contain communicationconnection(s) that allow communications with various other systems. Thecomputing systems also include one or more input devices, such as akeyboard, mouse, pen, voice input device, touch input device, etc., andone or more output devices, such as a display (including a touch-screendisplay), speakers, printer, etc. coupled communicatively to theprocessor(s) and computer-readable media via connections such as a bus.

Computer-readable media 304 stores computer-executable instructions thatare loadable and executable by one or more processor(s), as well as datagenerated during execution of, and/or usable in conjunction with, theseprograms. In the illustrated example, computer-readable media storesoperating system instances, which provide basic system functionality toapplications stored thereon. One or more of these components, includingthe operating systems, may be instantiated as virtual machines,containers, or as some other type of virtualized instantiation.

Processor(s) 302 may include one or more single-core processing unit(s),multi-core processing unit(s), central processing units (CPUs), graphicsprocessing units (GPUs), general-purpose graphics processing units(GPGPUs), or hardware logic components configured, e.g., via specializedprogramming from modules or application program interfaces (APIs), toperform functions described herein. In alternative examples one or morefunctions of the present disclosure may be performed or executed by, andwithout limitation, hardware logic components includingField-programmable Gate Arrays (FPGAs), Application-specific IntegratedCircuits (ASICs), Application-specific Standard Products (ASSPs),System-on-a-chip systems (SOCs), Complex Programmable Logic Devices(CPLDs), Digital Signal Processing unit(s) (DSPs), and other types ofcustomized processing unit(s). For example, a processing unit configuredto perform one or more of the functions described herein may represent ahybrid device that includes a CPU core embedded in an FPGA fabric. Theseor other hardware logic components may operate independently or, in someinstances, may be driven by a CPU. In some examples, examples of thecomputing systems may include a plurality of processing units ofmultiple types. For example, the processing units may be a combinationof one or more GPGPUs and one or more FPGAs. Different processing unitsmay have different execution models, e.g., as is the case for graphicsprocessing units (GPUs) and central processing units (CPUs).

Depending on the configuration and type of computing device used,computer-readable media include volatile memory (such as random accessmemory (RAM)) and/or non-volatile memory (such as read-only memory(ROM), flash memory, 3D XPoint, resistive RAM, etc.). Thecomputer-readable media can also include additional removable storageand/or non-removable storage including, but not limited to, SSD (e.g.,flash memory), HDD (Hard Disk Drive) storage or other type of magneticstorage, optical storage, and/or other storage that can providenon-volatile storage of computer-executable instructions, datastructures, program modules, and other data for computing systems.

Computer-readable media can, for example, represent computer memory,which is a form of computer storage media. Computer-readable mediaincludes at least two types of computer-readable media, namely computerstorage media and communications media. Computer storage media includesvolatile and non-volatile, removable and non-removable media implementedin any process or technology for storage of information such ascomputer-executable instructions, data structures, programming modules,or other data. Computer storage media includes, but is not limited to,phase change memory (PRAM), resistive RAM, 3D Xpoint non-volatilememory, static random-access memory (SRAM), dynamic random-access memory(DRAM), other types of random-access memory (RAM), read-only memory(ROM), electrically erasable programmable read-only memory (EEPROM),flash memory or other memory technology, compact disk read-only memory(CD-ROM), digital versatile disks (DVD) or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to storeinformation for access and retrieval by a computing device. In contrast,communication media can embody computer-executable instructions, datastructures, program modules, or other data in a modulated data signal,such as a carrier wave, or other transmission mechanism. As definedherein, computer storage media does not include communication media.

Various processes described herein are carried out as computingfunctions in conjunction with networking functions. For example, onecomputing device or system may cause transmission of a message toanother computing device via network interface hardware. This mayinclude, for example, passing by a software module a pointer, argument,or other data to a networking module. The pointer, argument or otherdata may identify data stored in memory or in a register that is to betransmitted to another computing device. The networking module mayinclude a protocol stack, and may read the data identified by thepointer, argument, or other data. The protocol stack may encapsulate thedata in one or more frames, packets, cells, or other data networkingprotocol structures. The protocol stack may call a network interfacedevice driver, to cause physical transmission of electrical, magnetic,or optical signals along a communication medium to a network element,such as a gateway, router, switch, hub, and so forth. An underlyingnetwork may route or switch the data to the destination. The destinationcomputing device may receive the data via a network interface card,which results in an interrupt being presented to a device driver. Aprocessor of the destination computing device passes the device driveran execution thread, which causes a protocol stack to de-encapsulate thedata in the packets, frames, and cells in which the data was received.The protocol stack causes the received data to be stored in a memory, aregister, or other location. The protocol stack may pass a pointer,argument, or other data that identifies where the received data isstored to a destination software module executing on the destinationcomputing device. The software module receives an execution thread alongwith the argument, pointer, or other data, and reads the data from theidentified location.

Network virtualization has existed for over a decade in the simplisticHost/Guest pattern, more commonly known as the Hypervisor (Host) and VM(Guest) components. The pattern assumes the Host is a physical systemmediating access to shared compute, network, and disc on behalf ofguests. Network accelerations from Host to Guest typically encompassstateless offloads—such as large segment offload (LSO), large receiveoffload (LRO), TCP Offload (TO) and stateful offloads, such as I/Ovirtualization (IOV)) offloads, and so forth. Stateless offloads aresoftware-mediated through the Virtual Switch (vSwitch) layers to theHost incurring a non-negligible processing cost at high density and athigh speeds, such as at 40 Gbps+ speeds. Stateful offloads completelybypass the host for network access yielding near native performance.

Both aforementioned offload models contain flaws that impact theirapplicability in Container Nested Network Virtualization (CNNV) models.In CNNV, the guest contains a virtual switch (vSwitch) that hostsmultiple host virtual network interface cards (virtual NIC or vNIC)and/or host NICs (hNICs). Containers have their own NIC,compartmentalized TCP/IP stack, and shared Kernel, thus appearing asconventional entities on the network. Containers are lightweight(relative to a VM), and thus can achieve densities of several orders ofmagnitude on a physical system.

Using a Nested vSwitch to address container networking has severaladvantages. Namely architectural correctness for OSI Layers, reusabilityof existing components, and future applicability to nested VMs. However,this does mean that the software processing cost for packet transfersfrom Miniport to hNIC endpoint is duplicated, potentially leading toscalability limitations.

The IOV model addresses the SW performance concerns but introducesundesired coupling points from Guest to HW, complicating operations suchas VM mirroring and migration. Since IOV bypasses the host, networkpolicy and security are bypassed, exposing the network fabric topotentially malicious guests.

In some embodiments, these issues are addressed with Synthetic PacketDirect (PD). The Synthetic PD model consists of a PD Provider Interface(PDPI) and PD Consumer Interface (PDCI). Under the PD model a PDPIpresents a queue for PDCI controlled Ethernet packet Post and Drainoperations. The vSwitch acts as a PDCI on the Host, consuming thenetwork Frames and converting to traditional specific NBLs beforedelivery to VMBus such that the VM ingress frame processing isunchanged.

With Synthetic PD, the Queues are projected from the Adapter on thePhysical Host directly into the Guest via a shared memory pluggable PDbuffer model. The end result is the adapter DMAs frames directly intothe Guest buffers of the hNIC associated with the Container. Frameprocessing occurs within the hNIC scope, thus the Host processing costsare circumvented (excluding optional network policies applied to headersvia Header Data Split (HDS)). Furthermore, the aforementioned IOVconstraints are also avoided since there is no HW coupling in the guest.The association of PD endpoint in the Guest to adapter PD queue ismediated through the shared memory buffer model. As such, mirroring andmigration operations are simplified.

Embodiments include a method to create, allocate, provision, and trackreceive queues based on MAC address and IP address. In some embodiments,the receive queues are created, allocated and tracked by the operatingsystem. Traffic is steered to containers based on one or more of: anaffinity to a specific MAC address when Containers are layer two (L2)addressable externally, an affinity to both destination MAC and IPaddress if Containers are bridged via MAC rewrites, where layer 3 (L3)is the container identifier, and an affinity based on a virtual network,a MAC address and an IP address.

Embodiments include a system comprising a software application in acontainer accessible by servers or clients to obtain information via anetwork I/O. One or more physical hosts have DPDK or PD capable MC. Oneor more VMs hosted on the physical host include one or more containers.A configuration store that contains a container queue information. Insome embodiments, the physical host is a computer, a kiosk, a mobiledevice, a game console or a multi-user, multi-application server.

Example Clauses

Example A, a system comprising: a local controller configured to:monitor resource utilization of the system; receive a policy templatethat includes one or more configurable network policy elements from acentral controller; calculate a network virtualization policy for atleast one virtualized computing resource implemented within one or morenodes associated with the local controller, the network virtualizationpolicy being based at least on the received policy template and themonitored resource utilization and the network virtualization policyconfiguring network connectivity for the at least one virtualizedcomputing resource; and distribute the network virtualization policy toone or more network infrastructure elements and/or to the one or morenodes; and a local store configured to store the monitored resourceutilization, the policy template, and the calculated networkvirtualization policy.

Example B, the system of Example A, wherein the local controller isfurther configured to request, based at least on the monitored resourceutilization and from the central controller, an updated policy templatethat indicates a change in the configurable network policy elements, thechange providing additional networking resources.

Example C, the system of Example A or Example B, wherein the at leastone virtualized computing resource includes a first virtual machineand/or a container executing within a second virtual machine.

Example D, the system of Example C, wherein the one or more configurablenetwork policy elements includes a plurality of sets of networkaddresses that are associated with each other based on a policyrelationship between the first virtual machine and the second virtualmachine, the local controller configured to allocate at least onenetwork address selected from the plurality of sets of network addressesto the at least one virtualized computing resource.

Example E, the system of Example C, wherein the one or more configurablenetwork policy elements includes a plurality of sets of securitypolicies that are associated with each other based on a policyrelationship between the first virtual machine and the second virtualmachine, the local controller configured to allocate at least onesecurity policy selected from the plurality of sets of security policiesto the at least one virtualized computing resource.

Example F, the system of Example C, wherein the one or more networkinfrastructure elements includes at least one virtualized networkfunction instantiated within the first virtual machine, the localcontroller configured to monitor and update a policy that is tailored tothe virtualized network function.

Example G, the system of Example C, wherein the local controller isconfigured to use one or more IP anchors to ensure connectivity to atleast one virtualized network function instantiated within the firstvirtual machine, the local controller configured to monitor and update apolicy that is tailored to the at least one virtualized network functionand that is associated with the one or more IP anchors.

Example H, the system of any one of Example A through Example F, whereinthe local controller is configured to use one or more IP anchors toensure connectivity to at least one node of the one or more nodes thathas received a policy update from the local controller.

Example I, the system of any one of Example A through Example H, whereinthe local controller is configured to provide discoverability servicesfor a virtual machine, a container, or a virtualized network functionthat has migrated to another system.

Example J, a method comprising: monitoring, by a local controller,resource utilization of a system; receiving, at the local controller andfrom a central controller, a policy template that includes one or moreconfigurable network policy elements; calculating, based at least on thereceived policy template and the monitored resource utilization, anetwork virtualization policy for at least one virtualized computingresource implemented within one or more nodes associated with the localcontroller, the network virtualization policy configuring networkconnectivity for the at least one virtualized computing resource; anddistributing, by the local controller, the network virtualization policyto one or more network infrastructure elements and/or to the one or morenodes.

Example K, the method of Example J, further comprising requesting, basedat least on the monitored resource utilization and from the centralcontroller, an updated policy template that indicates a change in theconfigurable network policy elements, the change providing additionalnetworking resources.

Example L, the method of Example J or Example K, wherein the at leastone virtualized computing resource includes a first virtual machineand/or a container executing within a second virtual machine.

Example M, the method of Example L, wherein the one or more configurablenetwork policy elements includes a plurality of sets of networkaddresses that are associated with each other based on a relationshipbetween the first virtual machine and the second virtual machine andwherein the method further comprises distributing, by the localcontroller at least one network address selected from the plurality ofsets of network addresses to the one or more nodes.

Example N, the method of Example L, wherein the one or more configurablenetwork policy elements includes at least a plurality of sets ofsecurity policies that are associated with each other based on a routingrelationship between the first virtual machine and the second virtualmachine, the method further comprising allocating at least one securitypolicy selected from the plurality of sets of security policies to theat least one virtualized computing resource.

Example O, the method of Example L, wherein the one or more networkinfrastructure elements includes at least one virtualized networkfunction instantiated within the first virtual machine, the methodfurther comprising monitoring and updating a policy that is tailored tothe virtualized network function.

Example P, the method of Example L, further comprising: using one ormore IP anchors to ensure connectivity to at least one virtualizednetwork function instantiated within the first virtual machine; andmonitoring and updating a policy that is tailored to the at least onevirtualized network function and that is associated with the one or moreIP anchors.

Example Q, the method of any one of Example J through Example O, furthercomprising using one or more IP anchors to ensure connectivity to atleast one node of the one or more nodes that has received a policyupdate from the local controller.

Example R, the method of any one of Example J through Example Q, furthercomprising providing discoverability services for a virtual machine, acontainer, or a virtualized network function that has migrated toanother system.

Example S, a system comprising: one or more processors;computer-readable media; programming instructions stored on thecomputer-readable media and executable by the one or more processors to:receive a policy template for a local environment that includes one ormore nodes, the policy template indicating one or more configurablenetwork policy elements allocated to one or more virtualized computingresources instantiated within the one or more nodes; determine, based atleast in part on the policy template, a network virtualization policyfor at least one virtualized computing resource instantiated within theone or more nodes, the network virtualization policy providingconnectivity to the at least one virtualized computing resource;distribute the network virtualization policy to one or more networkinfrastructure elements and/or to the one or more nodes; monitor networkresource utilization within the one or more nodes; and request anupdated policy template that indicates a change in the one or moreconfigurable network policy elements to provide additional networkresources allocated to the one or more virtualized computing resourcesinstantiated within the one or more nodes or to one or more newvirtualized computing resources instantiated within the one or morenodes.

Example T, the system of Example S, wherein the programming instructionsare further executable by the one or more processors to: receive theupdated policy template responsive to the requesting the updated policytemplate; and determine an updated network virtualization policy for oneor more of the at least one virtualized computing resource or a newvirtualized computing resource.

CONCLUSION

Although the techniques have been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the appended claims are not necessarily limited to the features oracts described. Rather, the features and acts are described as exampleimplementations.

All of the methods and processes described above may be embodied in, andfully automated via, software code modules executed by one or moregeneral purpose computers or processors. The code modules may be storedin any type of computer-readable storage medium or other computerstorage device. Some or all of the methods may alternatively be embodiedin specialized computer hardware.

Conditional language such as, among others, “can,” “could,” “might” or“may,” unless specifically stated otherwise, are understood within thecontext to present that certain examples include, while other examplesdo not include, certain features, elements and/or steps. Thus, suchconditional language is not generally intended to imply that certainfeatures, elements and/or steps are in any way required for one or moreexamples or that one or more examples necessarily include logic fordeciding, with or without user input or prompting, whether certainfeatures, elements and/or steps are included or are to be performed inany particular example. Conjunctive language such as the phrase “atleast one of X, Y or Z,” unless specifically stated otherwise, is to beunderstood to present that an item, term, etc. may be either X, Y, or Z,or a combination thereof. Any routine descriptions, elements or blocksin the flow diagrams described herein and/or depicted in the attachedfigures should be understood as potentially representing modules,segments, or portions of code that include one or more executableinstructions for implementing specific logical functions or elements inthe routine. Alternate implementations are included within the scope ofthe examples described herein in which elements or functions may bedeleted, or executed out of order from that shown or discussed,including substantially synchronously or in reverse order, depending onthe functionality involved as would be understood by those skilled inthe art. It should be emphasized that many variations and modificationsmay be made to the above-described examples, the elements of which areto be understood as being among other acceptable examples. All suchmodifications and variations are within the scope of this disclosure andprotected by the following claims.

What is claimed is:
 1. A computerized system comprising: one or morecomputer processors; and computer memory storing computer-useableinstructions that, when used by the one or more computer processors,cause the one or more computer processors to perform operationscomprising: allocating, by a central controller, a policy for a tenantnetwork; receiving, at the central controller and from a localcontroller associated with the tenant network, a declaration of a numberof containers hosted by one or more nodes or one or more virtualmachines of the tenant network; and sending, from the central controllerto the local controller and based at least in part on the declaration, apolicy template associated with the policy for the tenant network. 2.The system of claim 1, the operations further comprising: calculating,by the local controller, a network virtualization policy based at leastin part on the policy template; and distributing the networkvirtualization policy to the one or more nodes or the one or morevirtual machines of the tenant network.
 3. The system of claim 1,wherein the tenant network comprises at least one of a routing domainidentifier (RDID) or a host domain.
 4. The system of claim 1, whereinthe policy template indicates a change to one or more configurablepolicy elements.
 5. The system of claim 4, wherein the one or moreconfigurable policy elements comprise one or more of: Internet Protocol(IP) addresses, Media Access Control (MAC) addresses, port numbers, orcustomer addresses (CAs) for one or more routing domain identifiers(RDIDs).
 6. The system of claim 4, wherein the one or more configurablepolicy elements comprise load balancer virtual Internet Protocol (VIP)address to dynamic Internet Protocol (DIP) address mappings.
 7. Thesystem of claim 4, wherein: the one or more configurable policy elementscomprise constraints for service chain configuration; the service chainconfiguration comprises a path of service chain elements a data packettraverses during communication to or from a destination in a datacenter;and an individual service chain element comprises a load balancer, ananti-virus scanner, a firewall, or a packet inspection server.
 8. Thesystem of claim 4, wherein the one or more configurable policy elementscomprise local forwarding tables that include a destination with which anode or a virtual machine is able to communicate, wherein the localforwarding tables include encapsulate/decapsulate rules, network addresstranslation rules, or a range of IP addresses that are reachable by thenode or the virtual machine.
 9. One or more computer-storage mediahaving computer-executable instructions embodied thereon that, whenexecuted by a computing system having a processor and memory, cause theprocessor to perform operations comprising: allocating, by a centralcontroller, a policy for a tenant network; receiving, at the centralcontroller and from a local controller associated with the tenantnetwork, a declaration of a number of containers hosted by one or morenodes or one or more virtual machines of the tenant network; andsending, from the central controller to the local controller and basedat least in part on the declaration, a policy template associated withthe policy for the tenant network.
 10. The media of claim 9, theoperations further comprising: calculating, by the local controller, anetwork virtualization policy based at least in part on the policytemplate; and distributing the network virtualization policy to the oneor more nodes or the one or more virtual machines of the tenant network.11. The media of claim 9, wherein the tenant network comprises at leastone of a routing domain identifier (RDID) or a host domain.
 12. Themedia of claim 9, wherein the policy template indicates a change to oneor more configurable policy elements.
 13. The media of claim 12, whereinthe one or more configurable policy elements comprise one or more of:Internet Protocol (IP) addresses, Media Access Control (MAC) addresses,port numbers, or customer addresses (CAs) for one or more routing domainidentifiers (RDIDs).
 14. The media of claim 12, wherein the one or moreconfigurable policy elements comprise load balancer virtual InternetProtocol (VIP) address to dynamic Internet Protocol (DIP) addressmappings.
 15. The media of claim 12, wherein: the one or moreconfigurable policy elements comprise constraints for service chainconfiguration; the service chain configuration comprises a path ofservice chain elements a data packet traverses during communication toor from a destination in a datacenter; and an individual service chainelement comprises a load balancer, an anti-virus scanner, a firewall, ora packet inspection server.
 16. The media of claim 12, wherein the oneor more configurable policy elements comprise local forwarding tablesthat include a destination with which a node or a virtual machine isable to communicate, wherein the local forwarding tables includeencapsulate/decapsulate rules, network address translation rules, or arange of IP addresses that are reachable by the node or the virtualmachine.
 17. A computer-implemented method, the method comprising:allocating, by a central controller, a policy for a tenant network;receiving, at the central controller and from a local controllerassociated with the tenant network, a declaration of a number ofcontainers hosted by one or more nodes or one or more virtual machinesof the tenant network; and sending, from the central controller to thelocal controller and based at least in part on the declaration, a policytemplate associated with the policy for the tenant network.
 18. Themethod of claim 17, the operations further comprising: calculating, bythe local controller, a network virtualization policy based at least inpart on the policy template; and distributing the network virtualizationpolicy to the one or more nodes or the one or more virtual machines ofthe tenant network.
 19. The method of claim 17, wherein the tenantnetwork comprises at least one of a routing domain identifier (RDID) ora host domain.
 20. The method of claim 17, wherein the policy templateindicates a change to one or more configurable policy elements.